System and method for acquiring virtual and augmented reality scenes by a user

ABSTRACT

A preferred method of acquiring virtual or augmented reality (VAR) scenes can include at a plurality of locations of interest, providing one or more users with a predetermined pattern for image acquisition with an image capture device and for each of the one or more users, in response to a user input, acquiring at least one image at the location of interest. The method of the preferred embodiment can also include for each of the one or more users, in response to the acquisition of at least one image, providing the user with feedback to ensure a complete acquisition of the virtual or augmented reality scene; and receiving at a remote database, from each of the one or more users, one or more VAR scenes. One variation of the method of the preferred embodiment can include providing game mechanics to promote proper image acquisition and promote competition between users.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/668,915, filed Mar. 25, 2015, which is a continuation of U.S.application Ser. No. 13/302,977, filed Nov. 22, 2011, now issued as U.S.Pat. No. 9,017,163, which claims priority to U.S. Application No.61/448,133, filed Mar. 1, 2011 and U.S. Application No. 61/417,199,filed Nov. 24, 2010, the disclosures of which are incorporated herein intheir entirety by these references.

TECHNICAL FIELD

This invention relates generally to the virtual and augmented realityfield, and more specifically to a new and useful system and method foracquiring virtual and augmented reality scenes by a user.

BACKGROUND AND SUMMARY

Many techniques for capturing spherical images require complicatedequipment, including very expensive and specialized camera lenses orother fixtures that must be used. This in effect ensures that onlyhighly specialized and/or trained professionals to generate suchimagery. Collection of such data also typically requires speciallyequipped vehicles, which only increases the associated costs andspecialization required for capturing spherical image data. In somecases, image capture involves driving along substantially every road ofnearly every city to provide a satisfactorily large data set. Despitesuch investment in capturing this data, the resulting images are limitedto images taken from roads. Furthermore, maintaining the image datarequires constant collection, which again increases the costs associatedwith image acquisition. Despite these inherent limitations to the imagesavailable, numerous applications have been formed to utilize suchspatially associated imagery. In particular, augmented realityapplications on mobile devices have in recent years begun to find theimages to be a great resource. Yet, the complicated and costly imagecollection process ultimately limits the potential uses of this data.

A solution to the foregoing restrictions is described in detail below. Amethod of acquiring virtual or augmented reality (VAR) scenes accordingto a preferred embodiment can include: at a plurality of locations ofinterest, providing one or more users with a predetermined pattern forimage acquisition with an image capture device and for each of the oneor more users, in response to a user input, acquiring at least one imageat the location of interest. The method of the preferred embodiment canalso include for each of the one or more users, in response to theacquisition of at least one image, providing the user with feedback toensure a complete acquisition of the virtual or augmented reality scene;and receiving at a remote database, from each of the one or more users,one or more VAR scenes. One variation of the method of the preferredembodiment can include providing game mechanics to promote proper imageacquisition and promote competition between users. Other features andadvantages of the method of the preferred embodiment and its variationsare described in detail below with reference to the following drawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart depicting a method of acquiring virtual andaugmented reality scenes by a user in accordance with a preferredembodiment and a variation thereof.

FIG. 2 is a schematic diagram of a spherical image acquirable by a userin accordance with the method of the preferred embodiment.

FIG. 3 is a schematic diagram of an apparatus and/or user interfaceconfigured for acquiring virtual and augmented reality scenes by a userin accordance with the method of the preferred embodiment.

FIG. 4 is a schematic diagram of an apparatus and/or user interfaceconfigured for acquiring virtual and augmented reality scenes by a userin accordance with the method of the preferred embodiment.

FIG. 5 is a schematic diagram of a system for receiving and/or storingone or more virtual and augmented reality scenes acquired by one or moreusers in accordance with the method of the preferred embodiment.

FIG. 6 is a schematic diagram of a variation of the apparatus and/oruser interface configured for acquiring virtual and augmented realityscenes by a user in accordance with the method of the preferredembodiment.

FIG. 7 is a schematic diagram of user feedback displayable on a userinterface in accordance with the method of the preferred embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the inventionis not intended to limit the invention to these preferred embodiments,but rather to enable any person skilled in the art to make and use thisinvention.

As shown in FIG. 1, a method of the preferred embodiment can include ata location of interest, providing a user with a predetermined patternfor image acquisition with an image capture device in block S100 and inresponse to a user input, acquiring at least one image at the locationof interest in block S102. The method of the preferred embodiment canfurther include in response to the acquisition of at least one image,providing the user with feedback to ensure a complete acquisition of thevirtual or augmented reality (VAR) scene. The method of the preferredembodiment functions to reduce provide for a uniform, simple, andcomprehensive manner in which one or more users can collect image dataabout their respective surrounding environments. The method of thepreferred embodiment further functions to create uniformly accessibleVAR scenes, which can be retrieved from one or more users and providedto one or more viewers through network access to a remote database. Theimages acquired can be either two-dimensional or three-dimensionalimages, and they can be either still photographs or one or more framesof a video sequence.

Enabling capture of spatial imagery by common devices preferably furtherfunctions to enable widespread, crowd-generated, spatially assignedimage data. Since untrained users are expected to capture the images,the method preferably further functions to unobtrusively guide thecapture process so that novice photographers will be able to acquireusable quality VAR scenes. One preferred variation of the methoddescribed below utilizes game play mechanics to encourage proper captureof scene imagery. The game play mechanics preferably provide atask-based challenge for the user with device orientation as a centralcontrol mechanic. In the background of a game-like interaction, the gamepreferably results in the user properly orienting the phone for thecapture of a scene. Other non-game like interactions can alternativelybe used. Preferably, a user acquires images by directing the imagecapture device outward and systematically spins and directs the cameraat particular orientations as described below. In the end, an image of aspherical space is preferably created to simulate the world viewable inany direction from the perspective of the user.

As shown in FIG. 1, the method of the preferred embodiment can includeblock S100, which recites at a location of interest, providing a userwith a predetermined pattern for image acquisition with an image capturedevice. Block S100 preferably functions to present a user with one ormore cues, instructions, formats, and/or templates to follow inacquiring one or more of a series of images usable in a VAR scene.Preferably, the predetermined pattern does not contain undesirable“holes” or voids in the image data. The preferred predetermined patterncan be used to form a spatial image scene that can be described as asurface in the shape of a sphere, plane, cylinder, planar path, curvedplane, or any suitable surface. For example, if a full spherical imageis the objective, the predetermined pattern preferably guides the userto move the image capture device in a way that collects image data fromthe whole world around the user. The poles of a sphere or othersingularities can be special conditions that the predetermined patternincludes. As another example, if a user is capturing a plane of imagedata while walking, the predetermined pattern preferably ensures that afull plane worth of image data with no holes is captured. As usedherein, the preferred predetermined pattern can include aprogrammatically determined pattern that is subject to user override(i.e., through the user actuating the user interface as desired).Alternatively, some or all aspects of the preferred predeterminedpattern can be compulsory (i.e., through a feedback mechanism of thetype described below) to ensure proper acquisition of the image/s.

In another variation of the method of the preferred embodiment, thepredetermined pattern can include designated positions in which anindividual image or frame must be captured. When the image capturedevice is in position the image is captured from that designatedposition/orientation. In another variation, the preferred predeterminedpattern can contain rules for the physical and/or temporal spacing ofimages. For example, each individual image can require a change inposition and/or orientation that differs by some defined amount from atleast a second position and/or orientation. The second position and/ororientation preferably are the position and/or orientation of a secondimage. For example, if a first image is captured when the image capturedevice is directed in a North direction, the second image may not becaptured until the image capture device is oriented in a direction thatdiffers from the first image by a predetermined angular measure, i.e.,thirty degrees. The predetermined pattern can be presented asstraightforward instructions that are facilitated through visual, audio,or tactile feedback. In one example, the predetermined pattern canrequire the user to “paint the world,” where the user must collect imagedata for every desired region to “paint” (or “wipe clean” to reveal ascene or image). In this variation, the user is not directed to move toany particular orientation but the predetermined pattern indicates whatareas have been captured. By successfully “painting” the whole space,the user can complete the entire predetermined pattern. The guidance ofthe predetermined pattern can additionally be interactive andincorporate feedback and/or game mechanics as described in detail below.

Image acquisition preferably includes photographic and/or videoacquisition using a digital image capture device. A suitable imagecapture device can include a camera, a video camera, a laptop computer,a tablet computer, a smart phone, or any other handheld or mobile deviceconfigured to acquire photos or videos. Preferably, the image capturedevice can include at least one feedback channel for presenting thepredetermined pattern to the user, such as for example a display oraudio output that provides the user with the predetermined patternand/or ongoing feedback for image acquisition. Additionally, the imagecapture device preferably includes one or more embedded feedbackcontrollers to internally control the image acquisition process inresponse to the orientation and/or location of the device.

Preferably, the image capture device of the method of the preferredembodiment can be configured to determine its own orientation inthree-dimensional space. For example, the preferred image capture devicecan include one or more modules or sensors for detecting, determining,calculating, and/or providing a projection matrix, which is amathematical representation of an arbitrary orientation of athree-dimensional object having three degrees of freedom relative to asecond frame of reference. As an example, the projection matrix caninclude a mathematical representation of a device's orientation in termsof its Euler angles (pitch, roll, yaw) in any suitable coordinatesystem. Preferably, the image capture device can include one or morecameras (front/rear), an accelerometer, a gyroscope, a MEMS gyroscope, amagnetometer, a pedometer, a proximity sensor, an infrared sensor, anultrasound sensor, a global position satellite transceiver, WiFitransceiver, mobile telephone components, and/or any suitablecombination thereof for calculating the projection matrix and/or theassociated Euler angles. Orientation and/or position information can begathered in any suitable fashion, including device ApplicationProgramming Interfaces (API) or through any suitable API exposing deviceinformation, e.g., using HTML5 to expose device information includingorientation/location.

As shown in FIG. 1, the method of the preferred embodiment can furtherinclude block S102, which recites in response to a user input, acquiringat least one image at the location of interest. Block S102 preferablyfunctions to cause an image capture device to acquire, receive, select,and/or capture image data for processing into at least an image and/orat least a portion of a VAR scene. Preferably, if the image capturemechanism is still photography, then the acquisition of the at least oneimage is substantially simultaneous with a user input, such as a tactileinput or audio command. Alternatively, if the image capture mechanism isvideo photography, then the acquisition of the at least one image cancontinue for a duration following the user input. In anotheralternative, the acquisition of the at least one image can be initiatedin response to a predetermined orientation/location condition of theimage capture device, such that if the user positions/moves the imagecapture device in a predetermined fashion, the image capture device willautomatically acquire the image. In another alternative, the imagecapture mechanism can include both still photography and videophotography portions, in which case the image acquisition process canrespond as necessitated by the requirements of the image capture device.Another variation of the method of the preferred embodiment can includeacquiring a second image that substantially replicates a prior acquiredimage. Preferably, the second image acquisition can be performed inresponse to determining a low quality in the prior acquired image. Imagequality can preferably be determined at the image capture device, oralternatively at a remote database or server during a subsequent imageprocessing phase.

In another variation of the method of the preferred embodiment,acquiring at least one image can include acquiring at least one image ina floating point format to ensure full dynamic spectrum of the virtualor augmented reality scene. As noted below, the VAR scene can be anysuitable geometry, including for example a spherical image disposedsubstantially symmetrically about a nodal point. Acquisition of the VARimage data in a floating-point exposure preferably functions to allowthe user to view/experience the full dynamic range of the image withouthaving to artificially adjust the dynamic range of the images/scene.Preferably, the method of the preferred embodiment globally adjusts thedynamic range of each image as it is acquired such that a portion of theimage in the center of the display is within the dynamic range of thedevice. By way of comparison, high dynamic range (HDR) images appearunnatural because they attempt to confine a large image range into asmaller display range through tone mapping, which is not how the imageis naturally captured by a digital camera. Preferably, the method of thepreferred embodiment preserves the natural range of the image byadjusting the range of the image to always fit around (eithersymmetrically or asymmetrically) the portion of the image viewable in anapproximate center a display on which it is to be viewed. As notedelsewhere herein, the acquired image can include for example a video, aseries of still photographs, or any suitable combination thereof.Accordingly, the method of the preferred embodiment can further includeadjusting the floating point exposure of the acquired images in responseto any changes in the sequence of images that make up the VAR scene,such as for example adjustments in the lighting as a user progressesthrough a series of still images.

As shown in FIG. 1, the method of the preferred embodiment can furtherinclude block S104, which recites in response to the acquisition of atleast one image, providing the user with feedback to ensure a completeacquisition of the VAR scene. Block S104 preferably functions to guide,teach, instruct, and/or cause a user to acquire any subsequent images inaccordance with the predetermined pattern so as to ensure completion ofthe complete VAR scene. Block S104 preferably further functions toguide, instruct, direct, and/or control the image capture device tocapture images in a predetermined fashion to ensure a completeacquisition of the VAR scene. Preferably, the feedback can include atleast one of user-oriented feedback or device-oriented feedback.User-oriented feedback can include for example visual, tactile, or audiofeedback to communicate corrective suggestions to the user during theimage acquisition sequence. As an example, user-oriented feedback caninclude distorting a display and/or audio feed during motion of thedevice such that a user is prompted to wait until there is a clearimage, written instruction, sound, and/or the like prior to capturingthe intended image/frame. Preferably, acceleration of the device can bedetermined by any suitable sensor, such as the one or more cameras(front/rear), an accelerometer, a gyroscope, a MEMS gyroscope, amagnetometer, a pedometer, a proximity sensor, an infrared sensor, anultrasound sensor. In response to a device acceleration/motion, thedevice can preferably provide user-oriented feedback to the user inorder to get the user to manage the device motion and improve thecapture of the VAR scene.

Preferably, device-oriented feedback can include for example automatedcontrol of the timing, sequencing, and selection of captured stillimages/video frames based on one or more predetermined factors.Preferably, both user-oriented and device-oriented feedback is providedto the user. Alternatively, the device-oriented feedback can be directedto the image capture device with or without the user's knowledge.Preferably, the user-oriented feedback can be combined with one or moreaspects of the predetermined pattern noted above in block S100 of themethod of the preferred embodiment to provide the user with a continuousinput-feedback loop relating to the acquisition of the VAR scene suchthat the user does not stray or (per the device's action) is unable tostray from the predetermined pattern.

As noted above, the image capture device can preferably include one ormore cameras (front/rear), an accelerometer, a gyroscope, a MEMSgyroscope, a magnetometer, a pedometer, a proximity sensor, an infraredsensor, an ultrasound sensor—all or any of which can be used todetermine an acceleration/motion of the image capture device through thepredetermined pattern. Device-oriented feedback can preferably beinitiated in response to a sensor determination and/or a context of thepredetermined pattern (i.e., whether a sudden acceleration of the deviceproper at a particular point in the acquisition process.) Preferably,the method of the preferred embodiment provides device-oriented feedbackin response to an improper/non-ideal handling of the image capturedevice for a predetermined portion (context) of the predeterminedpattern. Device-oriented feedback can automate the selection of videoframes and/or the actuation of capturing still images, preferably inresponse to one or more predetermined conditions relating to the motionof the image capture device and/or the context of the predeterminedpattern.

As an example, device-oriented feedback can control the selection ofvideo frames and/or the actuation of capturing still images in responseto one or more of: an angular/linear distance between the presentlocation of the image capture device and the last captured/selectedimage/frame; an angular/linear distance between the present location ofthe image capture device and any other arbitrary previouslycaptured/selected image/frame in the predetermined pattern; a time sincethe last captured/selected image/frame; an exposure difference measuredfrom the last captured/selected image/frame; a current exposure level;an angular velocity of the image capture device; a shutter speed of theone or more cameras of the image capture device; a camera gain; adistance to any discrete guide point and/or guide region in thepredetermined pattern and/or location of interest; a product of theangular velocity of the image capture device (average/current) and theshutter speed of the one or more cameras of the image capture device;and/or any explicit instruction from the user indicating a relativeimportance of the frame/image currently capturable by the image capturedevice. Alternatively, the device-oriented feedback can include anyother usable measurements of the image capture device orientation and/orlocation, or any other suitable parameter relating to qualitative and/orquantitative aspects of the image sought.

In another variation of the method of the preferred embodiment, thepredetermined pattern can include at least one of a spatial pattern or atemporal pattern. Preferably, a spatial pattern and/or a temporalpattern can function to provide a user with a readily accessible andeasily understood representation of the predetermined pattern for imageacquisition. Example temporal patterns can include audible/visible cues(including musical cues) that follow a predetermined pattern indicatingwhen and/or in what direction the user should acquire the nextimage/frame. In another variation of the method of the preferredembodiment, the temporal pattern can include discrete intervals at whichthe user is signaled and/or instructed to capture the next image/framein the VAR scene. Alternatively, the temporal pattern can includediscrete intervals at which images/frames are automatically selected bythe image capture device with or without notification to the user. As anexample, if the image capture device includes a video camera, then thetemporal pattern can include one or more discrete intervals that pacethe acquisition of the series of frames in response to a predeterminedtiming, image capture device motion measurements, and/or image capturedevice input.

Suitable spatial patterns can generally include multi-sided enclosedgeometries and linear open geometries. As an example, the predeterminedpattern can include a path, trail, map or other one-dimensional routethrough which the user is instructed to acquire substantially planarimages. As shown in FIG. 2, another example spatial pattern can includea substantially spherical or spheroidal composite of images surroundinga nodal point 12. The spherical predetermined pattern 10 preferablyfunctions to permit a user and/or the image capture device to image anentire complete view of the space surrounding the nodal point 12.Preferably, the nodal point 12 is substantially coincident with a user,a user's head, or a portion of the user's head (i.e., a point betweenthe user's eyes). Alternatively, the nodal point 12 can be substantiallycoincident with the image capture device. As shown, the spherical image10 can define a series of latitudes 14 and a series of longitudes 16. Inone variation of the method of the preferred embodiment, thepredetermined pattern can include identifying points on the sphericalimage 10 by latitude 14/longitude 16 coordinates at which an imageshould be acquired by the user. Alternatively, the predetermined patternof the preferred embodiment can include instructions to the user to scana particular range of latitudes 14/longitudes 16 through each of aseries of longitudes 16/latitudes 14. Preferably, if the predeterminedpattern includes one or more images of a spherical image 10, then eachof the individual images/frames can be acquired and/or processed asspherical images by the image capture device. Similarly, if thepredetermined pattern includes a one-dimensional path, then each of theindividual images/frames can be acquired and/or processed as planarimages by the image capture device.

As shown in FIG. 3, a predetermined pattern and/or user feedback inaccordance with the preferred embodiment can be presented to the user ofthe device 20 through one or more visual cues. The device 20 of thepreferred embodiment can include a display 22, such as for example atouch screen display providing a user interface having one or moreactuators 24, and a front-facing camera 28. In the example embodimentdescribed above, if the user is instructed to acquire images/framescomposing a spherical image 10, then the device 20 can present apattern/feedback through visual cues, such as arrows 26 that direct theuser to move his or her device 22 in a predetermined direction.Preferably, the device 20 can use the front facing camera 28 todetermine a position of the user and/or the nodal point 12, as well asto determine a proper focal/imaging distance between the user and thedevice 20. As noted above, acquisition of a spherical image 10 caninclude a rotation of the device 20 through a range of latitudes 14(along longitudinal arrow 26) followed by a change in longitudes 16(i.e., pivot the device in the direction of latitudinal arrow 26).

As shown in FIG. 4, the predetermined pattern of the method of thepreferred embodiment can include instructions for rotating the device 20about one or more discrete axes 102, 104, 106. Preferably, theinstructions can be presented to the user through the display 22 of thedevice 20, although other suitable tactile and/or audio instructions canbe communicated in addition to or in lieu of visible instructions. Asshown in FIG. 4, the device 20 can preferably be rotated through aseries of pitch values about axis 102 in order to capture a series ofdiscrete images/frames 100, 100A, 100B. Preferably, the device 20 canadditionally and/or subsequently be rotated through a series of rollvalues along axis 106 for each series of images/frames captured 100,100A, 100B. Acquisition of a sufficient range of pitch values and rollvalues preferably results in the collection of an entire spherical image10 of the type described above. Alternatively, if the device 20 isequipped with front-facing and rear-facing cameras, then the sufficientrange of pitch values and roll values can be divided in half, as thedevice 20 will be capturing opposing segments of hemispherical images ateach acquisition. In another variation of the method of the preferredembodiment, the front-facing and rear-facing cameras can be configuredwith customized settings (i.e., exposure settings) such that each camerafunctions to supplement the other resulting in a fuller capture of allof the image data. Preferably, a user maintains the device 20substantially still along axis 106 and rotates the device 20 about axis106 during the image acquisition to ensure consistent and symmetricalimage acquisition. Accordingly, the device 20 can preferably include oneor more positioning functions, such as global position satellitecapabilities, WiFi triangulation capabilities, and/or mobile telephonetriangulation capabilities, in order to assist the user in maintainingthe device 20 substantially along axis 106 during the acquisitionprocess.

As shown in FIG. 6, another variation of the method of the preferredembodiment can include providing a map to guide the user along apredetermined route as at least part of the user-oriented feedback. Themap preferably functions to provide real-time or near-real time feedbackand/or predetermined pattern information to the user through the display22 of the device 20. The route 29 can include one or more predeterminedlocations at which the user is requested to acquire one or moreimages/frames for a VAR scene. In another variation of the method of thepreferred embodiment, a game system 90 can include pairing the device 20with a pedometer 92 to measure and/or direct a user's movement and/orprogress along the route 29. As noted above, the device 20 canpreferably include one or more positioning functions, such as globalposition satellite capabilities, WiFi triangulation capabilities, and/ormobile telephone triangulation capabilities, in order to assist the userin traversing the route 29.

As shown in FIG. 7, another variation of the method of the preferredembodiment can include providing a game to the user as part of theuser-oriented feedback. Preferably, the objective of the game can be toacquire virtual or augmented reality scenes. Providing a game to theuser preferably functions to motivate one or more users to acquire VARscenes while simultaneously ensuring that the acquired scenes arevaluable and viewable to one or more viewers. As shown in FIG. 7, oneexample game can include a map interface 200 showing a series ofcaptured VAR scenes 400 spread out with a series of open VAR scenes 402.Preferably, a point system or other suitable metric can be used tomeasure the value of each VAR scene 402, and a user with a predeterminednumber of points and/or ranking amongst other users can be in line forother compensation or prizes. Preferably, the competitive nature of thegame mechanics will cause a group of users can acquire a large set ofVAR scenes in a systematic and efficient manner, thereby providing VARmapping of entire geographical locations including neighborhoods orcities.

As shown in FIG. 1, another variation of the method of the preferredembodiment can include block S106, which recites receiving at a remotedatabase, from each of one or more users, one or more VAR scenes. BlockS106 preferably functions to aggregate, correlate, manage, store, and/ordistribute one or more VAR scenes to a series of viewers. Preferably,each of the one or more VAR scenes is acquired following the method ofthe preferred embodiment and variations thereof. Alternatively, one ofthe one or more VAR scenes can be acquired and/or created by other meansor methods and include non-image and non-video media. As shown in FIG.5, block S106 can further function to allow one or more users 30, 32, 34to transmit and/or upload their respective VAR scenes 40, 42, 44 to aremote database 50. The remote database 50 can be configured to includeand/or be connected to a server 60 that is connectable to one or moreviewers 80, 82, 84 through a network 70. Preferably, the server 60 canhost one or more directories, files, and/or web pages containing and/ordisplaying the one or more VAR scenes 40 42, 44. In such a manner, themethod of the preferred embodiment preferably functions to motivate andinstruct a series of untrained users to acquire VAR scenes of theirrespective environments and distribute those VAR scenes to any number ofinterested viewers 80, 82, 84.

In another variation of the method of the preferred embodiment, themethod can include compositing the at least one image and at least asecond image into the VAR scene. Compositing the at least two imagespreferably organizes and optionally optimizes the data for constructinga construct or data file for a spatial image scene. Preferably, thespatial scene can be designed for rendering with a 3D or 2D graphicsplatform such as OpenGL, WebGL, or Direct3D. The rendering canalternatively occur within a browser using HTML5 and/or CSS3 propertiesto apply appropriate transformations. In the HTML variation, HTML andCSS transforms can be used to render the VAR scene. As noted above, aVAR scene is preferably one that is optimally viewed withorientation/positioning parameters. For example, a viewer can preferablyexplore a spherical spatial image scene on a mobile device by rotatingand directing the mobile device in different directions. The displayedimage on the mobile device preferably corresponds to the view a viewerwould have seen during the capture of consecutive images. Additionally,pedometer sensing and/or GPS sensing can be used to navigate a spatialimage that incorporates the user position such as walking along a path.In the HTML rendering variation, the deviceorientation/location/displacement is fetched (e.g., through HTML5 or adevice API) and used to periodically update (e.g., 60 frames per second)the CSS transform properties of media of the virtual and augmentedreality view. Exploring a spatial image scene can alternatively beachieved on a desktop or fixed device.

In variations of the method of the preferred embodiment, compositing thetwo or more images can occur on the image capture device 20 or at theremote database 50. Preferably, the two or more images can be stitchedtogether using the ordering of the consecutive images and imageprocessing to align the two or more images. Additionally, orientationand/or location information that was collected at the image capturedevice while collecting the consecutive images is preferably used tospatially organize the images. Additionally, orientation and/or locationinformation and image processing techniques can be cooperatively used tocombine the consecutive images and assign the correct orientation toeach of the at least two images. For example, the at least two imagescan be stitched substantially from image processing techniques, but thenassigned a global position based on a GPS signal and an orientationbased on the orientation data (i.e., MEMS gyroscope, accelerometer, andthe like) provided by the image capture device. In another variation ofthe method of the preferred embodiment, audio and/or video datacollected can additionally be mapped to particular areas of the spatialimage scene. During viewing of the scene, the audio or video can bepresented chronologically to show where the user was pointing whilerecording the media, or alternatively, presented when viewing aparticular area.

Additionally, the method of a preferred embodiment can include directinga user to maintain a substantially constant viewing distance between theuser and the image capture device, which preferably functions toregulate the manipulation of the image capture device during movementthrough the predetermined pattern. The viewing distance is preferablydefined as the distance between the image capture device and the nodalpoint (e.g., the eyes of the user). The viewing distance canadditionally be defined as the focal length. A substantially constantviewing distance preferably creates a steady field of view during thetraversal of the predetermined pattern and aids in the capture of auniform scene of spatial imagery. An ideal viewing distance canadditionally be standardized to be within a particular range such asbetween one and two feet between the image capture device and the nodalpoint.

Preferably, a front facing camera can be used to detect the distancebetween the image capture device and the nodal point. As noted above,the nodal point can include a head, face, or other element of a user.The image capture device and the front facing camera preferablydetermine the viewing distance using the size of the element as anindicator. Preferably, the image capture device can track relativechanges in size of the element, or an approximation can be used based onthe average size of users' head, the spacing of the user's eyes, or anyother suitable measurement or approximation. Preferably, theuser-oriented feedback and/or device-oriented feedback can include realtime or near real time tracking of the substantially constant viewingdistance. For example, the image capture device can display text orother graphics that are best and/or only viewable at the substantiallyconstant viewing distance. Alternatively or additionally,device-oriented feedback can include adjusting camera parameters of theimage capture device in response to a non-compliant substantiallyconstant viewing distance. The device-oriented feedback can preferablyincludes adjusting viewing distance or field of view settings, includingfor example one or more physical settings of the camera (e.g., lensadjustments) or image processing techniques (e.g., adjusting cropping ofan image) based on the measured or approximated viewing distance.

In another variation, the method of a preferred embodiment can includeconstructing a scene of spatial imagery with parallax properties, whichpreferably functions to form three-dimensional images from imagescollected from differing viewpoints. During the image capture proceduredescribed above, useable parallax properties are preferably collecteddue to the viewing distance between the nodal object and the device (asopposed to a camera being rotated about an axis centered on the camera).The parallax properties can be inherent in the raw captured image dataor can be formalized in approximations of 3D model. The centers ofindividual images/frames are preferably weighted as having perpendicularviewing angles (i.e., plan view) and therefore not being influenced byparallax. By way of comparison, the sides of images/frames preferablycapture the parallax properties of how an object in the image/frameapparently shifts when viewed from different lines of sight. Parallaxproperties can preferably be used to construct stereoscopic imagery fromtwo different images/frames with corresponding parallax properties(i.e., viewed from opposing angles approximating the viewing angles ofthe human eyes).

Preferably, parallax properties can be constructed for individualobjects during the capture process by obtaining more imagery of thatobject from a plurality of perspectives. For example, to capture a focusobject, the method of the preferred embodiment can direct a user to movethe camera in a circular pattern around the object (or otherwise capturedifferent vantage points of the object) with the camera directedsubstantially at the object. Accordingly, when viewing the generated VARscene, a viewer can preferably view the focus object from a variety ofperspectives because of the enhanced capture process for the focusobject. Additionally, by capturing parallax properties, the distance ofobjects can be analyzed based on parallax shifts (near field objectsbeing more susceptible and far field objects less so). For example, edgedetection image processing can be used to isolate areas impacted byparallax in different ways. Additionally, parameterized parallaxproperties can be utilized in the display of the VAR scene, and imagesfrom the collected consecutive images that have a spatial separationsubstantially equal to the distance between human eyes can be used tocreate stereoscopic images for use in a VAR scene.

The image capture device and methods of the preferred embodiment can beembodied and/or implemented at least in part as a machine configured toreceive a computer-readable medium storing computer-readableinstructions. The instructions are preferably executed bycomputer-executable components preferably integrated with the device 20,the display 22, the remote database 50 and/or the server 60. Thecomputer-readable medium can be stored on any suitable computer readablemedia such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD orDVD), hard drives, floppy drives, or any suitable device. Thecomputer-executable component is preferably a processor but any suitablededicated hardware device can (alternatively or additionally) executethe instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

What is claimed is:
 1. A method comprising: receiving, from a firstclient device: a plurality of images of a location, and deviceorientation information for the plurality of images; compositing, usingthe device orientation information, the plurality of images into aspherical spatial image scene by spatially organizing the plurality ofimages into the spherical spatial image scene based on the deviceorientation information; and providing the spherical spatial image sceneto a second client device, wherein the spherical spatial image scene isexplorable by rotating or changing a direction of the second clientdevice.
 2. The method of claim 1, wherein compositing the plurality ofimages into the spherical spatial image scene comprises compositing theplurality of images surrounding a nodal point.
 3. The method of claim 2,wherein: the nodal point represents a position of a user of the firstclient device while the first client device captures the plurality ofimages; and the device orientation information comprises a detecteddistance between the first client device and the user of the firstclient device while the first client device captures the plurality ofimages.
 4. The method of claim 2, wherein compositing the plurality ofimages surrounding the nodal point comprises compositing the pluralityof images that include a complete view of space surrounding the nodalpoint.
 5. The method of claim 2, wherein the nodal point represents aposition of the first client device while the first client devicecaptures the plurality of images.
 6. The method of claim 1, wherein thedevice orientation information for the plurality of images comprisesgyroscope data from a gyroscope sensor of the first client device. 7.The method of claim 1, further comprising: determining parallaxproperties associated with each of the plurality of images; andweighting different portions of each of the plurality of images based onthe parallax properties to represent different viewing angles forobjects within the plurality of images.
 8. A non-transitory computerreadable medium storing instruction thereon that, when executed by atleast one processor, cause a system to: receive, from a first clientdevice: a plurality of images of a location, and device orientationinformation for the plurality of images; composite, using the deviceorientation information, the plurality of images into a sphericalspatial image scene by spatially organizing the plurality of images intothe spherical spatial image scene based on the device orientationinformation; and provide the spherical spatial image scene to a secondclient device, wherein the spherical spatial image scene is explorableby rotating or changing a direction of the second client device.
 9. Thenon-transitory computer readable medium of claim 8, wherein a viewableportion of the spherical spatial image scene changes according to arotation of the second client device.
 10. The non-transitory computerreadable medium of claim 8, further comprising instructions that, whenexecuted by the at least one processor, cause the system to compositethe plurality of images into the spherical spatial image scene in partby compositing the plurality of images surrounding a nodal point. 11.The non-transitory computer readable medium of claim 10, wherein thenodal point represents a position of the first client device while thefirst client device captures the plurality of images.
 12. Thenon-transitory computer readable medium of claim 10, further comprisinginstructions that, when executed by the at least one processor, causethe system to composite the plurality of images surrounding the nodalpoint in part by compositing the plurality of images that include acomplete view of space surrounding the nodal point.
 13. Thenon-transitory computer readable medium of claim 8, further comprisinginstructions that, when executed by the at least one processor, causethe system to composite the plurality of images into the sphericalspatial image scene in part by stitching the plurality of imagestogether using an ordering of the plurality of images as consecutivelycaptured by the first client device.
 14. The non-transitory computerreadable medium of claim 8, wherein the device orientation informationfor the plurality of images comprises gyroscope data from a gyroscopesensor of the first client device.
 15. A system comprising: at least oneprocessor; and at least one non-transitory computer readable storagemedium storing instructions thereon that, when executed by the at leastone processor, cause the system to: receive, from a first client device:a plurality of images of a location, and device orientation informationfor the plurality of images; composite, using the device orientationinformation, the plurality of images into a spherical spatial imagescene by spatially organizing the plurality of images into the sphericalspatial image scene based on the device orientation information; andprovide the spherical spatial image scene to a second client device,wherein the spherical spatial image scene is explorable by rotating orchanging a direction of the second client device.
 16. The system ofclaim 15, further comprising instructions that, when executed by the atleast one processor, cause the system to composite the plurality ofimages into the spherical spatial image scene in part by compositing theplurality of images surrounding a nodal point.
 17. The system of claim16, wherein: the nodal point represents a position of a user of thefirst client device while the first client device captures the pluralityof images; and the device orientation information comprises a detecteddistance between the first client device and the user of the firstclient device while the first client device captures the plurality ofimages.
 18. The system of claim 16, further comprising instructionsthat, when executed by the at least one processor, cause the system tocomposite the plurality of images surrounding the nodal point in part bycompositing the plurality of images that include a complete view ofspace surrounding the nodal point.
 19. The system of claim 15, furthercomprising instructions that, when executed by the at least oneprocessor, cause the system to composite the plurality of images intothe spherical spatial image scene in part by stitching the plurality ofimages together using an ordering of the plurality of images asconsecutively captured by the first client device.
 20. The system ofclaim 15, further comprising instructions that, when executed by the atleast one processor, cause the system to: determine parallax propertiesassociated with each of the plurality of images; and weight differentportions of each of the plurality of images based on the parallaxproperties to represent different viewing angles for objects within theplurality of images.